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Measuring the quality of VET using the Student Outcomes Survey 

Wang-Sheng Lee and Cain Polidano, Melbourne Institute of Applied 
Economic and Social research 



Currently, as noted in the review on the Australian vocational education and training (VET) system 
undertaken by the Organisation for Economic Co-operation and Development (OECD), there is a 
dearth of information available to students to help them make decisions about which course and 
provider will best meet their needs. 

This report by Lee and Polidano examined the potential use of information from the Student 
Outcomes Survey, including the use of student course satisfaction information and post-study 
outcomes, as a means of determining markers of training quality. This project was undertaken 
independent of the work of the National Centre for Vocational Education Research (NCVER) in 
this area and is a welcome complement to it. 

The main recommendation is that a ‘scoreboard’ approach of post-study outcomes is adopted as a 
means of measuring quality. The scoreboard would provide average outcomes by provider and field 
of education for a number of variables related to employment and training. While this approach has 
merit, it would necessitate a larger sample than that currently obtained for the Student Outcomes 
Survey in order for robust estimates to be provided. 

In addition to the scoreboard approach, the authors recommend other changes to the Student 
Outcomes Survey to ensure the data are better used. Coincidentally, NCVER has a number of 
projects currently underway that align with these recommendations: 

Publish individual provider information-. NCVER is reviewing the data protocols which currently 
proscribe the release of identified provider information. 

dr Collect more information on students and their labour market outcomes-. NCVER reviews the survey 
instrument regularly and welcomes Lee and Polidano’s suggestions. 

dr Cxpand the survey to include information on private feefor-service courses and all ACE (adult and community 
education) courses: NCVER has commenced a three-year project to address this data gap. 

dr Add a panel dimension to the survey-, others have also identified the need for longitudinal data that 
allow for the pathways of students to be tracked. The main issues with this proposal are the cost 
and the likely response rate in subsequent waves. 

In addition to the recommendations listed above, a further challenge now in the quest for greater 
transparency is to design a survey framework that applies across the entire tertiary sector. 



Tom Karmel 

Managing Director, NCVER 
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Executive summary 



The vocational education and training (VET) sector in Australia is moving towards a more 
competitive model of provision. In theory this will deliver more efficient outcomes by making 
training providers more responsive to the needs of students. However, in practice, realising such 
efficiency gains depends upon prospective students being able to determine how well each course 
meets their needs so that responsive providers are rewarded with higher demand for their courses. 
At present, there is litde information available for students to make such decisions, a key point 
raised in the review on the Australian VET system undertaken by the Organisation for Economic 
Co-operation and Development (OECD) (Hoeckel et al. 2008). Without information on the quality 
of training, there is the risk that providers will compete on fees alone, to the detriment of quality. 
Anderson (2005) found evidence of such ‘commoditisation’ occurring in the VET sector in 
response to market reforms. 

The aim of this report is to examine the potential use of information from the Student Outcomes 
Survey, including the use of student course satisfaction information and post-study outcomes, as a 
means of determining markers of training quality. In an analysis of the student course satisfaction 
measures, we found there are very small variations in reported average student satisfaction across 
providers, with and without controls for factors that differ among providers unrelated to training 
quality, such as differences in student intake. There are several possible reasons for this, including 
the sample used for the survey not being representative of all VET participants. 

We argue that outcome measures from the Student Outcomes Survey, such as further study and 
labour market outcomes, are more meaningful for students making choices on courses and 
providers, given that such outcomes are the main motivations for study. Further, differences in 
labour market outcomes also signal how valuable the skills acquired are to employers. All else being 
equal, the more favourable the graduate employment outcomes relative to competitors, the better a 
provider is in meeting the needs of students. 

We recommend the collation of outcome measures from the Student Outcomes Survey, along with 
other relevant course and provider information, to be made available as part of a ‘scoreboard’ of 
information on courses, similar to the Good universities guide for prospective higher education 
students. Such a depository of information makes it easy for students to compare and contrast 
courses and providers. However, we recognise that using outcomes for comparison has its 
drawbacks. In particular, differences in the outcomes across providers may not only reflect 
differences in quality, but also differences in the regions and in student clientele, which may create 
perverse incentives for providers to bias their student intake, shift their location, or pressure poor 
students to exit prematurely. For this reason we suggest that raw outcome measures are validated 
against measures that control for differences in student characteristics and student opportunities 
across providers, such as output from regression models. 

To ensure that data from the Student Outcomes Survey are better used, including as part of a 
‘scoreboard’ of information, we recommend a number of changes to the survey, listed in order, 
from what we consider to be easiest to hardest: 

•y" publish individual provider information 

•y" collect more information on students and their labour market outcomes 
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•v- increase the sample size and survey response rates 

■y- expand the survey to include information on private fee-for-service courses and all adult and 
community education (ACE) courses 

•y" add a panel dimension to the survey. 
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Introduction 



Reforms to the VET sector since the Hilmer report in 1993 have focused on making providers more 
responsive to the needs of students and employers. Central to these reforms have been legislative 
changes to allow non-TAFE (technical and further education) providers to compete on equal terms 
with TAFE for public funding of VET courses. These changes have given employers, together with 
employees, the right to choose the provider (public or private) of publicly funded training. 

While the impetus for the reforms may be to make the VET sector more responsive to training 
needs, evidence presented by Anderson (2005) shows that, while market reforms in the Australian 
VET system have led to cost reductions, they’ve come at the expense of quality. This outcome is 
consistent with economic theory which dictates that where there is poor information on product/ 
service quality, producers will typically respond to greater competition by reducing costs rather than 
improving services (Kranton 2003). An example of such behaviour is in the health sector, where, 
because of poor information on service quality, greater competition led hospitals in the United 
Kingdom to cut costs at the expense of quality (Propper, Burgess & Green 2004). 

The need to improve information on course quality, including better use of the Student Outcomes 
Survey, is a key recommendation of the OECD review on the Australian VET system (Floeckel et 
al. 2008). According to Floeckel et al. (2008), improving information on quality will enable 
prospective students to choose the course that best meet their needs. In this way, institutions are 
rewarded for the quality of their training and not just for reducing costs. 

The aim of this study is to address the OECD recommendation by examining possible uses by 
students for the Student Outcomes Survey data. The analysis is conducted in three key parts: first 
we evaluate the use of student course satisfaction measures in the survey. Given findings from the 
first part, in the second part we broaden the discussion by examining the information needs of 
students. In the final part, we make recommendations on the best mix of information from the 
Student Outcomes Survey and other sources, and how the survey can be improved. 



The role of quality measures in light of VET reforms 

The Australian Quality Training Framework (AQTF) formally defines quality as ‘the level of 
satisfaction with and effectiveness of vocational education and training organisations, their 
products and services, established through conformity with the requirements set by clients and 
stakeholders’ (ANTA 1999, p.22) and quality assurance as ‘the planned and systematic process of 
ensuring the consistent application of registration requirements by registered training organisations’ 
(ANTA 1999, p.23). 

It is, however, not simple to find a definition of quality in VET that suits all purposes. Quality in 
VET can mean different things to different people, as Gibb (2003, p.33) found from conducting 
focus group discussions. From the viewpoint of VET managers, ‘Quality is meeting the needs of 
the customer. It is concerned with levels of satisfaction and effectiveness. It is fitness for purpose; 
it is about achieving consistency.’ In contrast, quality to VET teachers is, ‘. . . that which makes 
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learning a pleasure and a joy. It is about excellence of the educational experience. It is about 
continuous improvement and finding ways to improve outcomes and satisfaction.’ 1 

At its essence, quality is a measurement of performance, and the range of definitions observed in 
the VET sector merely represents the various expectations against which people evaluate it. Using 
the examples above, VET teachers emphasise the processes of teaching, while VET managers 
emphasise meeting customer needs. How expectations vary depends on the interests of the 
individuals. In broad terms, we can identify three groups with varying interests in the quality of 
VET. First, there are prospective students, who, according to investment and consumption theories 
of education (Lazear 1977), are interested in the labour market returns and enjoyment derived from 
various courses. Second, there are employer interests, which include having adequate graduates 
available with the right mix of skills. Third, because they provide public funds to meet much of the 
cost of VET, governments are interested in ensuring that public funds are used efficiently to meet 
public interest objectives, including meeting both student and employer needs. 

While all three broad categories of interest in the VET sector are important, the focus of this study 
is on the development of quality measures to meet the interests of prospective students. Providing 
information on the quality of courses for students is important, given the recent and ongoing shifts 
towards a more ‘customer focused’ or ‘market-driven’ model of VET provision. Economic theory 
dictates that, in competitive markets where the quality of a good or service is hard to measure or 
where information on quality is poor, quality may be sacrificed in the pursuit of lowering costs 
(McMillan 2004; Kranton 2003). If customers have no or little information on the quality of goods 
or services, producers will not be rewarded for improving quality, only for reducing price. 

There is empirical evidence of this ‘commoditisation process’ in the literature, especially in the 
health sector (for example, Propper, Burgess & Green 2004) and findings from a report by 
Anderson (2005) suggest that reforms may have triggered such a process in the Australian VET 
sector. Reforms to VET in Australia during the 1990s were aimed at increasing competition in the 
sector; in particular, private providers were allowed to compete against public providers for 
government funding. These reforms led to a burgeoning in the number of private VET providers 
in Australia (Anderson 2005). However, despite the increase in contestability of government 
funding, Anderson (2005) in a study on the effects of market reforms concluded that the reforms 
of the 1990s had led to a reduction in the quality of education. Although Anderson did not point to 
the role of information in explaining poor market outcomes, this point was emphasised in the 
OECD review of the VET sector. Improving information on course quality, including better 
utilisation of the Student Outcomes Survey, was a key recommendation (Hoeckel et al. 2008). 
According to Hoeckel et al. (2008), improving information on quality will enable prospective 
students to choose the course that best meet their needs. 

The need for better information for students is even more pressing, given recent changes to the 
government funding model of VET that are likely to intensify competition. From July 2009, eligible 
students will be able to access a loan-type scheme similar to the Higher Education Contribution 
Scheme (HECS), known as VET FEE-HELP. Like HECS, the Australian Government will pay the 
course fees (from approved providers), with students repaying part of the course costs after course 
completion. 2 For providers, this means that, instead of applying directly to the government for 
funding, they must compete for students. For eligible students, this means that they have greater 
freedom to choose a course and provider that best meets their needs. However, if greater freedom 
of choice is to translate into better outcomes, students need accurate information on the quality of 
available courses across providers. 



1 No focus groups involving students were conducted in the Gibb (2003) study. 

2 While it does not cover accommodation and general living expenses, it allows students to be able to borrow funds to 
help pay for all or part of their tuition fees. Students approved for VET FEE-HELP assistance will have a loan with 
the Australian Government which will, on the student’s behalf, pay their tuition fees to their approved VET provider. 
At the time of writing of this report, VET FEE-HELP loans can only be accessed through approved VET providers; 
this list of approved VET providers is still growing. 
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An overview of the 
Student Outcomes Survey 



The Student Outcomes Survey is an annual, nationally stratified, randomly selected sample of VET 
completers. In essence it is a satisfaction and destination survey of those exiting the VET system 
upon completion. VET completers are identified as those who in the previous year were awarded a 
qualification (graduates) or who successfully completed part of a VET course and then left the 
VET system (module completers). It excludes those who undertook leisure or recreation short 
courses. Although the survey has been conducted since 1999, because we are interested in 
comparing student perceptions across provider types we limit our sample to the last four waves, 
2005 to 2008. Information on provider types other than TAFE is unavailable prior to 2005. 

A point to keep in mind is that the sample size of the Student Outcomes Survey fluctuates between 
81 000 (plus ‘top ups’ by individual providers) and 300 000 every alternate year (table 1). This 
allows for national and state reporting to be carried out every year, with institute reporting in 
alternate years (the large sample years). According to NCVER, while the sample size varies from 
year to year, the survey methodology remains unchanged (NCVER 2005, 2006, 2007, 2008). 



Table 1 VET enrolments and Student Outcomes Survey sample and responses 





Enrolments 1 


Sample 2 


Responses 3 


Response rate (%) 


2005 


1 606 400 


300 425 


100 904 


39 


2006 


1 650 800 


95 228 


36 663 


39 


2007 


1 676 000 


300 1 04 


96 633 


32 


2008 


1 665 000 


107 294 


39 300 


37 


Total 


6 598 200 


803 051 


273 500 


34 



Notes: 1 Number of people in VET in the year previous, National VET Provider Collection, 2007. 

2 Number of people who received a survey in the year after completing. 

3 Survey returned and completed. 



At present, due to current data protocols, only information on the ‘type’ of provider is publicly 
available. Provider type may be categorised as TAFE, other government, ACE or private. The ACE 
sector provides locally accessible and community-based educational opportunities for adults, 
especially for people who may have difficulty participating in more formal education groups 
(NCVER 2009). For the purposes of this study, we merge TAFE and other government providers 
(such as agricultural colleges) because there are only a small number of observations for the latter. 

A limitation of the Student Outcomes Survey is the low survey response rates (table 1), which are 
around 40% for graduates and around 30% for module completers over the period of interest. 
Response rates across the provider types are roughly comparable, although graduates from private 
providers report around a 5% lower response rate. Low response rates means that the sample may 
not be representative of graduate and module completers, because survey respondents are known 
to have different characteristics from non-respondents. 

The issue of non-response bias has been raised in survey evaluation reports. In the 2008 report (I- 
VIEW 2008), post-survey analysis of non-respondents found that they differed significantly from 
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respondents in a number of ways. Some of the key differences between (self-completion and 
telephone) respondents and non-respondents found in the 2008 survey are: 

Non-respondents are more likely to be male. 

•y" For TAFE graduates, non-respondents are less likely to report employment as their main 
motivation for training. 

•y" Non-respondents are more likely to report achieving their main motivation for training. 

•y" Non-respondents rated general skills and learning experiences of their courses more highly than 
respondents, but their experiences of teaching and assessment less highly. 

Many of these differences have been found in prior surveys. Of particular importance to this study 
is the finding of no estimated statistical differences in reported overall course satisfaction between 
respondents and non-respondents. 

Issues of sample size aside, the data in the Student Outcomes Survey are not representative of all 
VET courses because there is no information on students who completed fee-for-service courses 
from private providers 3 or from those who dropped out of the course without completing a 
module (estimated by Callan [2005] to be about 16% of all VET students), and information on 
students from ACE providers is limited to those from Victoria and New South Wales. 4 Further, 
there is less information available from ACE courses because ACE students are given a shorter 
questionnaire. For example, ACE students are not asked questions on post-training outcomes such 
as enrolment in further study, employment and wages. 

It became evident after the analyses for this report were completed that the coding of the ‘language 
other than English’ (LOTE) variable differed for the 2005 cohort. Further, missing values were 
included in the descriptive analysis of the ‘mode of delivery’ variable. Standard practice excludes 
missing values. However, both are inconsequential to the overall result. 



Descriptive statistics 

As background, we present a snapshot of key respondent and provider characteristics, including 
reporting main motivations for training, provider characteristics and individual characteristics. 

Main motivations 

From table 2, we can conclude that, across all fields of education, the main motivations and 
characteristics of VET participants are varied. From the data, we cannot generalise to say that some 
fields of education have more homogenous training needs than any other. While there are various 
reasons for undertaking training given across all fields, labour market reasons (find work, upskill, 
reskill and general skills) dominate. When considering the reported motivations, it is pertinent to 
keep in mind that they were elicited six months after ceasing study, so it is possible that responses 
may be influenced by post-study outcomes. 



3 This is not limited to the Student Outcomes Survey but the National VET Provider Collection is used as the sample 
frame. 

4 Due to an error, Victorian fee-for-service students from private providers were inadvertendy included in the 2005 
survey. These data are omitted because there is insufficient information to make robust comparisons. 
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Table 2 Main motivation for studying and student characteristics by main field of education 
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Provider and student characteristics 

A key consideration when using quality measures is to establish whether differences in the measures 
reflect differences in the quality of the courses or other differences unrelated to quality. In 
comparisons across providers, differences in quality measures unrelated to the quality of training 
might be differences related to the providers themselves (such as the types of courses offered and 
their location) and/ or differences in the students. The statistics in tables 3 and 4 show the average 
characteristics of providers and their students respectively. 

From table 3 we can conclude that the split between rural and urban training is roughly the same, 
with each provider type servicing both in approximately equal proportions. There are more 
noticeable differences between the types of courses undertaken; in particular, and to be expected, 
ACE providers are more likely to deliver non-certificate training in subject enrolments. The level of 
the course also appears to be tied to funding, with government-funded courses more likely to be 
higher level (certificate III/IV and above) and fee-for-service courses being more elementary. 
Courses delivered through ACE are more likely to involve personal instruction and, because they 
tend to be more elementary, the duration of these is also typically shorter. 



Table 3 Characteristics of VET provision by provider type and funding source 





TAFE & other govt 

Govt- Fee-for- 

funded service 1 

o/ o/ 

/o /o 


Govt- 

funded 

o/ 

/o 


ACE 

Fee-for- 

service 

% 


Private 

Govt- 

funded 

% 


Location 












City 


52 


47 


42 


45 


55 


Rural 


42 


47 


56 


53 


41 


Remote 

Qualification level 


6 


6 


2 


1 


5 


Diploma and above 


18 


5 


1 


2 


2 


Certificate III & IV 


50 


28 


26 


26 


58 


Certificate 1 & II 


24 


19 


24 


24 


32 


Other certificate 


3 


35 


4 


4 


1 


Statement of attainment/subject only 

Field of education 


4 


13 


46 


43 


6 


Natural and physical sciences 


1 


0 


0 


0 


0 


Information technology 


4 


2 


4 


3 


1 


Engineering and related technologies 


17 


17 


1 


1 


21 


Architecture and building 


6 


6 


0 


0 


4 


Agriculture, environmental and related 


6 


5 


1 


2 


4 


Health 


4 


9 


2 


5 


2 


Education 


3 


10 


5 


9 


2 


Management and commerce 


22 


14 


14 


13 


35 


Society and culture 


14 


7 


15 


16 


11 


Creative arts 


5 


1 


1 


1 


1 


Food, hospitality and personal services 


8 


11 


5 


6 


12 


Mixed-field programs 


8 


15 


8 


6 


4 


Subject only enrolment 
Continued over page 


0 


2 


44 


39 


3 
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TAFE & other govt 




ACE 


Private 




Govt- 


Fee-for- 


Govt- 


Fee-for- 


Govt- 




funded 


service 1 


funded 


service 


funded 




% 


% 


% 


% 


% 


Modes of course delivery 2 

Personal/classroom communication 
with instructor(s) 


61 


73 


78 


81 


59 


Printed materials 


42 


41 


43 


43 


45 


Web-based resources 


14 


9 


9 


8 


9 


Videotape, CD or DVD 


18 


17 


16 


16 


13 


Online communication with 
instructor and other students 


6 


4 


4 


3 


4 


Radio 


1 


0 


1 


0 


0 


Television 


3 


2 


2 


2 


2 


Video conference 


2 


1 


1 


1 


1 


Teleconference 


1 


0 


0 


0 


1 


Workshop practical activities 


29 


29 


25 


27 


29 


Other 


5 


4 


4 


3 


7 


Not stated 


1 


1 


1 


1 


1 


Course hours 












Less than 50 hours 


31 


66 


63 


67 


39 


50-199 hours 


38 


18 


18 


16 


34 


200-540 hours 


18 


10 


15 


14 


22 


More than 540 hours 


9 


3 


4 


3 


4 


Missing 


3 


4 


0 


0 


1 


Number of respondents 


1 69 892 


66 238 


4 023 


1 092 


25 037 



Notes: 1 Fee-for-service is training for which most or all of the cost is borne by the student or a person or organisation on 
behalf of the student. 

2 Respondents could select more than one response; hence, totals add up to more than 100%. Missing values 
included in the analysis 
Source: Student Outcomes Survey, 2005-08. 

As with the variation in student training needs (table 2), there are differences in respondent 
characteristics across provider types (table 4). Respondents from ACE institutions are more likely 
to be from non-English speaking backgrounds, are older, female, are less likely to hold post-school 
qualifications and are motivated more by the need to develop general skills than respondents from 
other provider types. Another point of difference is that those who completed government-funded 
courses with private providers are much more likely than respondents undertaking fee-for-service 
or government-funded TAFE and other government courses to be trainees or apprentices (48% 
compared with 7% and 18% respectively). 
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Table 4 Respondent characteristics by provider type and funding source 





TAFE 

Govt- 

funded 

% 


Fee-for- 

service 

% 


Govt- 

funded 

% 


ACE 

Fee-for- 

service 

% 


Private 

Govt- 

funded 

% 


Speak a language other than 


58 


57 


68 


76 


62 


English at home 












Age 












15-19 


21 


16 


8 


10 


25 


20-24 


21 


8 


7 


5 


23 


25^14 


35 


40 


35 


34 


30 


45-64 


21 


34 


42 


44 


21 


65+ 


1 


2 


8 


7 


1 


Has a disability 


11 


8 


9 


7 


9 


Main reason for training 












Get a job 


22 


9 


16 


13 


17 


Voluntary upskill 


26 


32 


24 


29 


32 


Compulsory upskill 


14 


29 


13 


17 


27 


Reskill 


11 


6 


7 


7 


7 


General skill 


14 


13 


24 


19 


10 


Other 


13 


11 


16 


15 


7 


Apprentice/trainee 


18 


7 


n.a. 


n.a. 


48 


Highest prior education 












Diploma/advanced diploma/bachelor 


18 


27 


18 


24 


11 


Certificate III & IV 


20 


22 


10 


11 


15 


Certificate 1 & II 


13 


7 


2 


2 


16 


Other cert. 


11 


11 


3 


2 


13 


Year 12 


17 


11 


24 


22 


18 


Less than Year 12 


21 


22 


44 


39 


27 


Male 


44 


53 


28 


27 


44 


Work while studying 


42 


61 


n.a. 


n.a. 


58 


Number of respondents 


169 892 


66 238 


4 023 


1 092 


25 037 



Notes: na = not collected from students of ACE providers. Voluntarily upskill is those who report undertaking training to get 
a better job or promotion, to get extra skills for current job, to develop existing business or to start a new business. 
Compulsory is those who report undertaking training as a job requirement. Reskill is those who report undertaking 
training to try for a different career. Other is those who report undertaking training to get into another course of study, 
to get skills for community/voluntary work or to increase confidence and self-esteem. 

Source: Student Outcomes Survey, 2005-08. 



In summary, from the data presented in tables 2 to 4, we can conclude that the training needs, 
measured as motivations for training and respondent characteristics, are varied and that these 
variations extend across all fields of education. In contrast to the variety of training needs, the 
courses offered by provider types are relatively uniform, as are the characteristics of respondents 
who attend each. However, we note that these aggregate data mask differences that are likely to be 
present across individual providers. 
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Student perceptions of quality 



Broadly speaking, quality measures from the Student Outcomes Survey can be categorised as 
information on student perceptions of course quality and student post-study outcomes. In this 
section, we examine the use of student perceptions of course quality. An advantage of using student 
perceptions of quality over using outcomes is that the former allows for easy comparison of quality, 
regardless of the motivation for studying. For instance, while wage progression or post-study 
employment outcomes may be appropriate measures for students who undertake training to gain a 
promotion or employment, respectively, such outcomes are not appropriate for those who 
undertake a course for pleasure or to help prepare them for further training. 



Producing course satisfaction instruments 

In the Student Outcomes Survey, information on student satisfaction is elicited by asking 
participants to respond to statements on three aspects of their course — teaching, assessment and 
general skill and learning experiences (see table A1 in the appendix). These opinions are reported 
on a five-point scale, where one is ‘strongly disagree’ with the statement and five is ‘strongly agree’ 
with the statement. After being asked to respond to statements on quality, respondents are asked to 
respond on the same five-point scale, to the statement, ‘Overall, I was satisfied with the quality of 
this training’. 

Given the sequence in which respondents are asked to respond to these statements, it is assumed 
that respondents first form an impression of each of the three course aspects and then combine 
these impressions, together with other relevant information, to decide on a level of course 
satisfaction. However, reverse causation is a possibility; that is, individuals form a view of course 
satisfaction first and use it to form impressions on aspects of course quality. If this is the case, then 
it would be pointless to use impressions of course aspects because they wouldn’t provide any 
information on quality that could not be gleaned by examining overall satisfaction. 

While testing for reverse causation is very difficult, we can test whether there is or is not more than 
one factor driving responses to statements on the various aspects of quality. If there is only one 
underlying factor (such as overall course satisfaction) driving response to all statements, then 
conducting separate analysis on the variables driving responses on each aspect of quality is 
unnecessary. Results from a principal components analysis show that there is sufficient variation in 
responses to conclude that there are three underlying factors driving responses and not one. 5 
Further, results show that statement responses under each aspect of quality are linked to the same 
underlying factor, which suggests that responses are reliable, or internally consistent. From these 
results, we can conclude that it is worthwhile examining responses to each of the three course 
aspects and, because statement responses under each aspect are linked to the same underlying 
factor, they can be combined to form three scales, one for each aspect. 

Producing a scale involves weighting and combining responses to each statement on a course 
aspect for each individual. While there are many different ways of weighting the statements, 



5 The three factors are extracted using an Eigenvalue of 1, which is standard in the literature (de Vaus 2002). 
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without a clear reason for choosing one way over another, we elect to give each item the same 
weight and we simply take the average of all responses. 

The mean scores, across all individuals in the sample, are presented in table 5, along with the mean 
overall rating of satisfaction. What is clearly noticeable from table 5 is that, across all fields of study, 
teaching is rated most highly and general skill and learning experiences is rated the lowest. Another 
interesting point is that the average overall course satisfaction lies between the highest and lowest 
aspect rating for all fields of study, which may suggest that individuals are combining the quality 
ratings when forming an overall view of course satisfaction. A final point is that the differences in 
means among fields of study are small, albeit significant. 6 



Table 5 Mean ratings for aspects of course quality on a scale of 1 to 5, by field of education 





Teaching 


Assessment 


General skill 
& learning 
experiences 


Overall 

satisfaction 


Natural and physical sciences 


4.23 


4.14 


3.94 


4.16 


Information technology 


4.18 


4.05 


3.77 


4.00 


Engineering and related technologies 


4.30 


4.12 


3.84 


4.13 


Architecture and building 


4.24 


4.06 


3.85 


4.10 


Agriculture, environmental and related studies 


4.37 


4.12 


3.81 


4.18 


Health 


4.37 


4.18 


3.84 


4.19 


Education 


4.36 


4.22 


3.77 


4.13 


Management and commerce 


4.26 


4.13 


3.86 


4.09 


Society and culture 


4.33 


4.18 


3.99 


4.17 


Creative arts 


4.27 


4.06 


3.90 


4.11 


Food, hospitality and personal services 


4.37 


4.16 


3.85 


4.23 


Mixed-field programs 


4.32 


4.09 


3.90 


4.17 


Subject only enrolment 


4.41 


4.08 


3.77 


4.21 



Source: Student Outcomes Survey, 2005-08. 



We present the distribution of the mean scores for the three aspects of course quality — teaching, 
assessment and general skill and learning experiences — together with the distributions of overall 
satisfaction ratings in table 6. 

The descriptive statistics in table 6 show that across all measures there are only minor differences 
between provider types. The only differences of note are the higher teaching ratings for ACE 
providers. For instance, around 70% of those who attended an ACE institution had a mean 
teaching score of between 4.1 and 5 (out of 5), compared with 67%, 63% and 65% of fee-for- 
service TAFE, government-funded TAFE and government- funded private providers, respectively. 
However, caution should be exercised when interpreting descriptive statistics, as any differences in 
satisfaction between provider types may be masked by differences in the characteristics of students 
who attend these institutions and differences in the nature of the training offered. In the section 
below, we use multivariate analysis to estimate the differences in student satisfaction across 
providers after controlling for such differences. 



6 Results from a t-test suggest that many of the differences between fields of study are significantly different at the 5% 
level. 
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Table 6 Distribution of mean ratings for aspects of course quality 





TAFE 




ACE 




Private 




Govt- 


Fee-for- 


Govt- 


Fee-for- 


Govt- 




funded 


service 


funded 


service 


funded 




% 


% 


% 


% 


% 


General skill and 
learning experiences 












1. 0-2.0 


2 


4 


3 


4 


4 


2.1-3. 0 


9 


14 


11 


11 


11 


o 

ri 

1 

CO 


47 


51 


52 


55 


45 


4. 1-5.0 


41 


31 


35 


30 


41 


Assessment 












1. 0-2.0 


2 


2 


2 


2 


2 


2.1-3. 0 


6 


6 


6 


7 


7 


o 

ri 

1 

CO 


44 


45 


44 


42 


42 


4. 1-5.0 


49 


47 


48 


48 


48 


Teaching 












1-2 


1 


1 


1 


2 


2 


2.1-3 


4 


3 


3 


3 


4 


3.1-4 


32 


29 


25 


24 


30 


4.1-5 


63 


67 


71 


70 


65 


Overall satisfaction 












1 (strongly disagree) 


2 


2 


2 


2 


3 


2 


4 


3 


3 


5 


4 


3 


8 


7 


5 


6 


8 


4 


50 


51 


47 


44 


47 


5 (strongly agree) 


36 


37 


42 


42 


38 


Number of respondents 


170 553 


66 226 


1 088 


4 025 


25 095 



Note: Columns may not sum to 1 00% because of rounding error. 

Source: Student Outcomes Survey, 2005-08. 



Multivariate analysis 

While it is possible to adjust measures of student satisfaction to account for differences in factors 
unrelated to training quality by merely re-weighting the data, this is a difficult task if there are 
differences in multiple factors, which is most often the case. To control for multiple sources of 
difference, the best method is to use multivariate analysis. Multivariate analysis allows us to remove 
the influence of observed differences that are unrelated to course quality, such as differences in the 
characteristics of students across providers and differences in the courses offered. The remainder is 
assumed to represent the differences in student satisfaction that are related to the quality of the 
courses offered by providers. 

We estimate separate ordinary least squares (OLS) regression models for each of the satisfaction 
measures — mean scores for teaching, learning experiences and general skills, assessment and overall 
course satisfaction — using all waves of the sample. Applying OLS regression means that we treat 
each of the dependent variables as a cardinal measure of satisfaction: someone who reports a 
course satisfaction of five out of five is assumed to be twice as happy with their course as someone 
who scores a two. Studies using similar scales, such as five-point measures of life satisfaction, have 
shown that whether the dependent variable is treated as a cardinal measure (OLS) or as an ordered 
measure (ordered probit) has little effect on the results (see for example, Stutzer & Frey 2006). An 
advantage of treating all of the dependent variables as cardinal is that the model results are easily 
comparable. 
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The variables included in the models are chosen for the purpose of trying to test differences in 
course satisfaction across provider types. Therefore, variables that are not derived for ACE 
providers, such as whether the course was part of a traineeship or apprenticeship are not included. 
Excluded is also the dummy variable for whether the respondent completed the course or not 
(module completers or graduates) because it is potentially endogenous; that is, individuals fail to 
complete the course because they were dissatisfied with the quality of the training. 7 We also exclude 
variables of training delivery modes because they should be captured in the effects of provider 
types. That said, we test to see whether differences in provider types are related to differences in 
delivery modes by examining the effect the inclusion of these variables has. 

An issue with undertaking this type of analysis is that there are likely to be unobserved personal and 
course-related variables that affect both course satisfaction and explanatory variables. For example, 
whether an individual studied part-time or full-time is not observed in the Student Outcomes 
Survey, but is likely to be linked with course satisfaction and individual characteristics. Controlling 
for all sources of unobserved individual and course heterogeneity is difficult. Due to the lack of 
year-on-year variation in the data from the survey (due to the small changes in year-on-year 
characteristics of students and course undertaken), there is no point estimating results from a fixed- 
effects estimation using a constructed pseudo panel model of the Student Outcomes Survey. 8 

Model results 

The estimated model coefficients and t-statistics for each of the student satisfaction models are 
presented in table 7. 

Because all of the explanatory variables are dummy (categorical) variables, the estimated 
coefficients represent the value of the dependent variable (mean score for teaching, mean score for 
assessment, mean score for general skill and learning experiences and overall satisfaction with the 
course) for an otherwise average individual on a scale of 1 to 5, when the given category is present, 
relative to when the reference category is present. For example, from the first column in table 7, the 
0.02 estimated coefficient for fee-for-service courses at a TAFE institute can be interpreted as 
follows: on average, respondents who had undertaken a fee-for-service course with a TAFE rated 
the training at 0.02 points higher on a scale of 1 to 5 than those who undertook a government- 
funded course at a TAFE institute. In percentage terms, fee-for-service training at a TAFE institute 
is rated 0.4 percentage points higher than government-funded training undertaken at a TAFE 
institute. 9 The t-statistics reported in the cells next to the estimated coefficients is a measure used to 
test whether the estimated model coefficient is different from zero. The greater its absolute value, 
the more confident we are that the coefficient is different from zero. Significance at 5% and 10% 
are indicated by a double or a single asterisk respectively. 



7 When we included a binary variable of whether the respondent was a module completer (1 if module completer and 0 
for graduate) in the models we observed consistently large negative coefficients with an unusually small standard error, 
a good indication that the variable is endogenous in the satisfaction equations. 

8 Using fixed-effects panel data approaches estimates relationships in the data, based only on changes in the average 
group characteristics and group satisfaction levels over time. In this way, differences in course content and unobserved 
differences in individuals across the fields of study have no effect on the results. 

9 Calculated as 0.02 divided by 5 and multiplied by 100. 
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Table 7 Estimated coefficients for the OLS regression models of satisfaction 





Teaching 


Assessment 


General skill 
& learning 
experiences 


Overall 

satisfaction 




coeff. 


t-stat 


coeff. 


t-stat 


coeff. 


t-stat 


coeff. 


t-stat 


Constant 


4.18 


643.67" 


4.05 


587.82" 


3.94 


539.55" 


4.12 


494.81" 


Course provider and funding source 

TAFE government-funded (reference case) 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


TAFE fee-for-service 


0.02 


6.00** 


-0.02 


-3.90** 


-0.07 


-14.51** 


0.01 


2.85** 


ACE fee-for-service 


0.02 


0.71 


-0.06 


-1.72* 


-0.04 


-1.10 


-0.02 


-0.59 


ACE government-funded 


0.07 


3.89** 


-0.04 


-2.08** 


0.01 


0.53 


0.03 


1.19 


Private government-funded 

Age 


0.03 


5.14** 


-0.03 


-5.65** 


-0.01 


-0.93 


-0.02 


-3.22** 


15-19 (reference case) 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


20-24 


0.02 


3.24** 


0.02 


4.60** 


0.01 


2.52** 


0.00 


0.16 


25-29 


0.07 


12i9*. 


0.07 


1 1 .44** 


0.01 


0.87 


0.03 


3.38** 


30-34 


0.10 


16.47** 


0.10 


15.69** 


0.01 


2.00** 


0.06 


7.08** 


35-39 


0.11 


18.57** 


0.11 


17.22** 


0.01 


2.03** 


0.07 


9.19** 


40-44 


0.12 


20.32** 


0.11 


16.83** 


0.01 


1.58 


0.07 


8.99** 


45-49 


0.12 


20.71** 


0.11 


16.59** 


-0.02 


-2.31** 


0.06 


7.91** 


50-54 


0.13 


20.23** 


0.11 


15.68** 


-0.03 


-3.45** 


0.08 


9.04** 


55-59 


0.14 


19.11** 


0.12 


15.39** 


-0.01 


-1.68* 


0.11 


11.01** 


60-64 


0.18 


17.31** 


0.12 


10.98** 


0.01 


1.11 


0.13 


9.60** 


654- 


0.22 


16.48** 


0.11 


8.10** 


0.12 


7.99** 


0.18 


10.81** 


Male 


-0.02 


-4.60** 


-0.04 


-11.64** 


-0.08 


-21.24** 


-0.02 


-5.26** 


Born in a non-English speaking country 


-0.06 


-12.75** 


-0.05 


-10.43** 


0.12 


22.03** 


-0.02 


-4.07** 


Speak a language other than English 
at home 


0.02 


4.41** 


0.02 


4.39** 


-0.02 


-4.59** 


0.01 


1.89* 


Field of education 


















Management and commerce 
(reference case) 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


Natural and physical sciences 


0.03 


1.40 


0.01 


0.63 


-0.01 


-0.64 


0.04 


1.49 


Information technology 


-0.08 


-9.71** 


-0.07 


-8.40** 


-0.06 


-7.08** 


-0.11 


-10.41** 


Engineering and related technologies 


0.04 


7.91** 


0.00 


0.91 


0.01 


1.03 


0.02 


2.53** 


Architecture and building 


0.00 


0.61 


-0.04 


-5.04** 


0.02 


2.97** 


0.00 


-0.26 


Agriculture, environmental and 
related studies 


0.07 


10.92** 


-0.03 


-4.13** 


-0.02 


-2.34** 


0.03 


3.19** 


Health 


0.03 


4.73** 


0.00 


-0.17 


0.06 


7.04** 


0.01 


1.15 


Education 


0.05 


6.68** 


0.06 


7.01** 


0.02 


1.98** 


0.03 


2.67** 


Society and culture 


0.08 


15.00** 


0.03 


5.30** 


0.10 


16.74** 


0.07 


10.38** 


Creative arts 


0.04 


4.65** 


-0.08 


-9.92** 


0.01 


1.47 


0.02 


1.91* 


Food, hospitality and personal services 


0.11 


18.44** 


0.04 


6.75** 


0.02 


2.69** 


0.09 


12.22** 


Mixed-field programs 


0.02 


3.41** 


-0.03 


-3.91** 


0.09 


12.59** 


0.01 


1.20 


Subject only enrolment 


-0.02 


-1.02 


-0.06 


-3.50** 


0.01 


0.54 


-0.03 


-1.48 


Location 


















Urban (reference case) 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


Regional 


0.04 


1 1 .47** 


0.04 


12.18** 


0.05 


13.08** 


0.04 


10.79** 


Remote 


0.04 


6.41** 


0.05 


7.38** 


0.10 


13.63** 


0.07 


9.07** 


Region missing 


0.02 


1.95* 


0.03 


2.35** 


0.02 


1.71* 


0.02 


1.51 


Continued over page 
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Teaching Assessment General skill Overall 

& learning satisfaction 

experiences 





coeff. 


t-stat 


coeff. 


t-stat 


coeff. 


t-stat 


coeff. 


t-stat 


Level of course 


















Diploma/advanced diploma 


-0.12 


-22.00** 


-0.08 


-13.47** 


0.06 


9.31** 


-0.13 


-18.68' 


Certificate III & IV 


-0.06 


-15.68** 


-0.02 


-4.81** 


0.03 


7.21** 


-0.09 


-17.07' 


Certificate 1 & II (reference case) 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


Other certificate 


0.01 


1.56 


-0.03 


-4.87** 


-0.03 


-3.98** 


0.01 


0.75 


Subject only or statement of attainment 


0.05 


6.90** 


0.04 


5.79** 


0.00 


-0.59 


0.09 


9.54' 


Length of course 


















Less than 50 hours (reference case) 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


50-199 hours 


0.02 


4.90** 


0.04 


9.22** 


0.07 


16.08** 


0.03 


6.35' 


200-540 hours 


0.06 


12.74** 


0.10 


20.94** 


0.14 


28.06** 


0.10 


17.23' 


More than 540 hours 


0.09 


13.71** 


0.11 


16.56** 


0.17 


24.77** 


0.15 


18.19' 


Hours missing 


0.04 


4.96** 


0.06 


5.98** 


0.12 


11.66** 


0.07 


6.15' 


Main reason for studying 


















Get a job (reference case) 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


Upskill 


0.05 


1 1 .25** 


0.05 


10.57** 


-0.06 


-12.68** 


0.03 


5.29' 


Reskill 


0.02 


3.28** 


0.04 


6.67** 


-0.06 


-8.57** 


-0.01 


-1.44 


Requirement of job 


0.00 


-0.18 


-0.02 


-3.87** 


-0.22 


-40.96** 


-0.05 


-8.91' 


General skill 


0.03 


5.96** 


0.02 


3.83** 


-0.11 


-18.17** 


0.02 


3.43' 


Other 


0.04 


6.93** 


0.03 


5.24** 


-0.08 


-13.18** 


0.02 


3.16' 


Has a disability 


-0.05 


-9.82** 


-0.07 


-12.86** 


-0.04 


-7.98** 


-0.05 


-8.31' 


Survey year 


















2005 (reference case) 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


2006 


0.01 


1.20 


-0.01 


-2.00** 


-0.01 


-1.43 


0.00 


0.31 


2007 


0.03 


7.59** 


0.00 


0.75 


0.01 


2.35** 


0.01 


2.02' 


2008 


0.05 


10.10** 


0.03 


4.66** 


0.04 


7.65** 


0.04 


5.91' 


Highest prior qualification 


















Higher education 


-0.04 


-6.52** 


0.00 


0.74 


-0.33 


-50.52** 


-0.13 


-17.33' 


Diploma/advanced diploma 


-0.03 


-5.13** 


0.02 


3.16** 


-0.17 


-24.70** 


-0.09 


-11.55' 


Certificate III & IV 


-0.02 


-3.34** 


0.02 


3.50** 


-0.07 


-12.75** 


-0.05 


-8.11’ 


Certificate 1 & II 


-0.02 


-3.43** 


0.01 


1.22 


-0.01 


-1.99** 


-0.02 


-3.57' 


Other certificate 


-0.01 


-2.37** 


0.00 


0.08 


-0.03 


-5.33** 


-0.03 


-4.41' 


Completed secondary school 


-0.02 


-4.97** 


0.00 


0.06 


-0.10 


-18.78** 


-0.05 


-7.27* 


Didn't complete secondary school 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 


ref. 



(reference case) 



Note: ** Significant at 5%, ‘Significant at 1 0%. 

Source: Student Outcomes Survey, 2005-08. 



Overall, the magnitude of the model results in table 7 is consistent with the findings from the 
descriptive statistics; that is, there are statistically significant differences between respondent ratings 
of provider types, but the magnitudes of these differences are so small that it is hard to make any 
strong claims over the relative quality of one against another. For example, we cannot rule out the 
possibility that these differences may be spuriously linked to unobserved differences in respondent 
characteristics across provider type that are unrelated to the quality of the training, such as whether 
the training was taken part- or full-time. 

As for differences in provider types, the model results in table 7 show only minor differences in the 
quality scores across individual characteristics, with few differences of note. In terms of magnitude, 
the largest coefficient across all of the explanatory variables is -0.33 points on a scale of 1 to 5, or a 
7-percentage point difference in the relationship between holding a pre-training higher education 
qualification relative to no qualification on the mean ‘general skills and learning experiences’ score. 

There may be several explanations for the small differences in quality observed above. 
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1 The small differences may be because those who may have been least satisfied with the quality 
of the education are not included in the survey — those who ceased studying without completing 
at least a module, estimated to be around 16% of all VET students (Callan 2005). 

2 Student perceptions of quality may not be valid instruments of actual training quality. One 
reason why they may not be representative of actual course quality is that the survey is 
conducted around six months after completion, at which time students may have forgotten 
aspects of the course. 

3 The results presented above may be right: there may in fact be high satisfaction with training 
courses with little variation in course quality between provider types. This may be because the 
Australian Quality Training Framework, which is a set of national training standards for 
provider registration, actually ensures a uniformly high training quality. 

We cannot rule out the first point because there is no information in the Student Outcomes Survey 
about those who did not complete at least a module, or the circumstances under which they ceased 
studying. Given that these individuals are estimated to comprise around 16% of all VET students, 
not an insignificant number, priority should be given to an analysis of the characteristics and 
experiences of those who cease studying before completing a module. It needs to be noted, 
however, that the survey was designed to measure outcomes of students who completed training. 

On the second point, it is difficult using the Student Outcomes Survey data to test whether 
perceptions of quality is a valid measure, mainly because perceptions of quality are measured at the 
same time as the outcomes, so that perceptions themselves may be affected by the outcomes. This 
point is underlined by regression results when an indicator for module completers is included in the 
model. Results from this model produced strong evidence to suggest that the module identifier was 
an endogenous variable in course satisfaction ratings. This suggests that there is a two-way (non- 
causal) relationship between course satisfaction and module completion: students become module 
completers (cut short their course) because of low satisfaction with their course, but also exiting 
study without completion may lower their perception of the course six months after completing. 
Note, however, that in VET many students do not intend to complete qualifications. They may exit 
the system when they have acquired the skills they sought. Ideally, validation would take place by 
assessing the skills of a sample of students before and after completing the training and examining 
how their skill accumulation is linked to satisfaction with their course. 

On the third point, the lack of individual provider data and data from private provider fee-for- 
service courses in the Student Outcomes Survey means that we cannot test this hypothesis. Data at 
the provider type level are highly aggregated and mask possible differences in student satisfaction 
across individual providers. 

Putting aside the first two points, even if we could conclude that the small differences in perception 
of quality across provider types are due to the uniformly high standard of training, there are still more 
fundamental questions that may be considered in relation to the use and design of quality measures: 

V If there is little variation in training quality between provider types, is there a need for quality 
measures? 

V If so, who would use the measures and for what purpose? What outcomes will these measures 
help to achieve? 

V What are the potential user-information needs? For example, do prospective students need 
information on provider types or is information at an institutional, campus, course or module 
level more important? Do they need information on facets other than the quality of training, for 
example, on-campus amenities and services? 

V Are these needs already met from other data sources? If not, how well does the Student 
Outcomes Survey meet these needs currently? What changes would be needed to the survey to 
accommodate them? 

These questions are addressed in the remainder of this report. 
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What information do students 
need to make good choices? 



The study so far has examined the use of student course satisfaction measures from the Student 
Outcomes Survey to evaluate quality. Results from this analysis suggest that prospective students 
would gain little insight from such measures. Just as good information can improve market 
outcomes, bad information can lead to poor outcomes. This raises the question, ‘What information 
do students need to make good choices?’ 

To answer this question, we need to consider why students engage in VET. From a theoretical 
perspective, students are deemed to engage in education to improve their future labour market 
prospects (investment model) and/or because they derive pleasure from studying (consumption 
model) (Duncan 1976). Under the human capital model, an individual’s choice of course and 
provider depends on whether their discounted future benefits outweigh the costs of studying. 

Future benefits are typically identified as being from the financial (higher future wages) and non- 
financial (greater job satisfaction and securing expanded employment options) benefits. Costs 
involved in studying are immediate and may include course costs (such as fees, transport and 
equipment costs) and foregone income and time that could have been spent in leisure, socialising or 
with family. When weighing up the future benefits and costs, because of time preference people 
tend to discount the importance of future benefits. In contrast, the consumption model emphasises 
the pleasure that individuals derive from undertaking a course at a particular institution, which may 
depend on the content of the course, campus amenities and on-campus activities, plus 
characteristics of the surrounding area. 

From the Student Outcomes Survey data (table 8), it is clear that investment motivations are 
important. From those surveyed between 2005 and 2008, the vast majority, across all fields of 
education, report investment motivations as the primary reason for undertaking a VET course. 10 In 
the majority of cases, students are said to be motivated to find work, improve their skills or to reskill. 
However, the emphasis on investment motivations does vary across fields of study, with around 22% 
of arts and 27% of mixed-field students reporting other motivations being of primary importance. 



10 It should be kept in mind that the reported motivations for studying are elicited six months after course completion and 
may be influenced by the student outcomes. Ideally, this information would be collected at the time study is undertaken. 
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Table 8 Main reason for studying 





Sci. 


IT 


Eng. 


Arch. Agr. & 
env. 


Hlth 


Educ. Man.& 
com. 


Soc. 

& 

cult. 


Arts 


Hosp. 


Mix. 


Sub. 

only 




% 


% 


% 


% 


% 


% 


% 


% 


% 


% 


% 


% 


% 


Find work 


27 


21 


16 


22 


11 


12 


9 


19 


21 


19 


23 


15 


12 


Voluntary upskill 


20 


27 


29 


26 


34 


31 


36 


36 


24 


18 


24 


17 


25 


Compulsory upskill 


15 


4 


32 


29 


25 


29 


26 


14 


15 


2 


23 


9 


21 


Reskill 


17 


8 


6 


8 


8 


13 


11 


9 


14 


15 


8 


5 


5 


General skill 


9 


26 


9 


7 


13 


6 


8 


14 


11 


24 


12 


28 


20 


Other 


14 


14 


7 


6 


10 


9 


11 


8 


15 


22 


10 


27 


17 


No. of respondents 


1 502 


9 430 


46 443 16 077 15 474 13 293 12 974 58 149 32 917 10 066 24 115 24 517 


4 525 



Source: Student Outcomes Survey, 2005-08. 



An important point to note is that, while investment motivations dominate, there are a range of 
desired labour market outcomes that may be of interest to prospective students. From table 8, it 
appears that there is a broad spread of labour market motivations, including finding work, career 
progression (voluntary upskill and general skills) and a change of career (reskill). This indicates that 
students may be interested in a range of different labour market outcomes from VET, including 
wage progression, employment prospects and the types of jobs graduates find. 

The strong emphasis on employment motivations for studying suggests that students would value 
information on post-study employment outcomes highly in making decisions on courses and 
providers. Another argument for using outcomes is that they also reflect the value that employers 
place on the skills acquired. All else being equal, for a given field of education, the greater the 
average wage and employment rates from a provider, the more closely the skills developed from the 
course meet the needs of employers. Although employment outcomes are the prime motivation, 
other motivations, such as to gain access to another course, are still important. Therefore, other 
outcomes, such as enrolment in further education, are also important information to students. The 
Student Outcomes Survey does include this type of information. 

To derive measures from the Student Outcomes Survey for student use, a number of issues need to 
be resolved. First, while there is information on provider type (that is, TAFE, other government, 
ACE and private provider), there is no publicly available information on individual providers. 
Second, information on key labour market outcomes is only collected six months after completion, 
with limited information on the longer-term outcomes. Third, more information is needed on 
employment outcomes after completion. Fourth, to evaluate course outcomes across provider 
types, more contextual information needs to be available, adjusted for differences in students across 
courses and providers. 

These issues arise because the Student Outcomes Survey was not designed for the purpose of 
evaluating courses across providers; rather, it was designed to produce broad sector-wide indicators 
for public accountability purposes. One of the main uses of the survey is for deriving annual 
indicators for publication in the Annual National Report of the Australian VET System to address 
two (out of six) key performance measures (ANTA 2004). These two key performance measures are: 

•y" the level of student employment outcomes and benefits after training and their satisfaction with 
their training program 

•y" the extent to which Indigenous Australians engage with and achieve positive outcomes from 
vocational education and training. 
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Using and developing 
quality measures from the 
Student Outcomes Survey 



The focus of this section is on the use and development of the Student Outcomes Survey to derive 
quality measures for students to use in making decisions on courses and providers. However, we 
point out that developing such measures does not discount their use for public accountability 
purposes. In fact, some of the recommendations to develop the survey dataset will assist in public 
reporting, such as reporting on the benefits of VET required under the Annual National Report 
and against Council of Australian Governments (COAG) objectives under the National Agreement 
for Skills and Workforce Development (Council of Australian Governments 2009). In developing 
Student Outcomes Survey data for use by students, it is pertinent to examine possible uses for the 
data. Below are a few examples of how similar data are used in other sectors for the same purpose. 

A first point of comparison is the information provided to prospective higher education students in 
Australia in the Good universities guide. The Good universities guide provides students with an overview 
of information on institutes, their campuses and the courses offered (including information about 
VET providers). More importandy, it provides a ‘scoreboard of comparisons’ on fees, entry 
requirements and course outcomes across all institutes where a particular course is offered. Course 
outcomes includes the percentage of graduates in employment in the year after completion, the 
destination of graduates, starting salaries and student course satisfaction ratings. 

A similar ‘scoreboard’ approach has already been adopted for VET in British Columbia in Canada 
(BCStats 2008) by using information from the Diploma, Associate Degree and Certificate Student 
Outcomes (DASCO) Survey. As well as including information on student characteristics (including 
student work and study history, gender, age and reasons for enrolling), the publication includes 
information on key graduate outcomes, including satisfaction with various aspects of the course, 
overall satisfaction with course, employment rates and wage rates and hours worked (see appendix 
B for an example of a scoreboard for the Justice Institute of British Columbia). Unlike the 
scoreboard approach in the Good universities guide, the approach used for VET in British Columbia 
only includes outcomes for the institutes as a whole by comparison with the average across all 
providers in British Columbia and no comparisons are made at the course level. 11 

Recommendation 1 : Adopt a scoreboard of post- study outcomes to measure quality 

A scoreboard of study and post-study outcomes from the Student Outcomes Survey would help 
Australian VET students make better decisions on the best course for their needs, which in turn 
would reward providers who best meet these needs. Such a scoreboard may include average 
outcomes by provider and field of education for: 

dr the main reason for doing the training 

dr proportion taking further study and types of further study 

dr current employment and earnings compared with before study 

dr occupation and industry of employment after study and job satisfaction 

dr relevance of training to their post-study job. 



11 Hoeckel et al. (2008) also provide examples in the US states of Arkansas and West Virginia, where information on the 
quality and effectiveness of VET programs is available for public review. 
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To provide robust estimates of such variables by course and provider would require a larger sample 
than that currently obtained for the Student Outcomes Survey. 

Accompanying outcome information with relevant course and provider information (similar to the 
Good universities guide) will help students more easily make comparisons. Course information may 
include content, course fees and course length, while provider information may include campus 
amenities and services, transport and information on governance, such as participation and awards 
received from the AQTF voluntary Excellence Criteria program. 

Because it takes time to gain a true picture of any labour market benefits from education and 
training, any outcomes should be measured over time. For example, for those who are already in 
employment when they undertake VET, the employment benefits may not be transmitted, in the 
form of higher wages and/ or increased job retention, until years after completion. Similarly, 
employment rates measured in the first year after graduation may reflect the efforts of providers to 
find their graduates work placements, which may not necessarily reflect the quality of the training. 

A criticism of using outcome measures is that they do not take into account differences in the 
characteristics of students who graduate from different courses. For example, there is much 
criticism of average school exam results published each year in the United Kingdom because clearly 
the performance of schools is not just a measure of the quality of the education, but a measure of 
the ability of the school graduates (Goldstein & Myers 1996). 

Such criticisms are relevant when comparing outcomes across individual VET providers. There are 
marked differences in ability and employment and education history among VET students. In 
response to increasing competition, individual providers, especially small private providers, are 
specialising in meeting the needs of specific clientele (Ferrier, Dumbrell & Burke 2008). 

Segmentation in training markets means that differences in training outcomes across providers is 
likely to be influenced not only by the quality of the training, but also by the characteristics of the 
students. For example, using raw post-training employment rates as a measure of quality will over- 
inflate the performance of providers who specialise in providing ‘in-house’ training to company 
employees, compared with providers who specialise in delivering courses to new school graduates. 

If differences in average outcomes among providers are not valid measures of quality — driven by 
differences in region or graduate characteristics rather than differences in course quality — then such 
indicators may reduce the benefits from providing students with quality information. In particular, 
it would reward providers who take the students with the best employment prospects, which may 
lead them to bias their student intake, shift their location, or pressure poor students to exit 
prematurely. Therefore, methods of analysis that control for the characteristics of the student 
cohort must be employed. 

Recommendation 2: Validate the use of post-study outcomes as a measure of quality 

Validating the use of the measures will provide confidence that labour market outcomes are a 
measure of course quality, which will ensure that students choose the best courses and that providers 
of the best courses are rewarded. Validating the use of average labour market outcomes involves 
testing the extent to which differences in average outcomes are due to differences in course quality, 
as opposed to differences in other factors unrelated to quality, such as differences related to students 
and local opportunities available to students. A simple way of doing this would be to tabulate the 
average characteristics of students across providers (for given courses) and employment rates in the 
regions where providers operate. This is a crude approach, but it may highlight whether more 
detailed analysis is warranted. If more detailed analysis is needed, regression analysis would be the 
best approach. Regression analysis allows the effects of provider type on labour market outcomes to 
be measured, independent of a range of observed student and regional characteristics that may also 
be significant. However, a downside of this approach is that it is more time-consuming and would 
require more information to be collected on respondents in the Student Outcomes Survey, such as 
information on their work and education history. If average employment outcomes are not valid 
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measures of course quality, then the use of more sophisticated regression techniques may be the only 
way of producing outcome measures that are comparable across providers. 

An alternative, but less attractive approach to validating outcomes, would be to include contextual 
information on the provider’s region and their graduate characteristics into the scoreboard (which 
is the approach used for comparing VET providers in British Columbia). For example, the average 
unemployment rate may give students a measure of the employment conditions in the region in 
which the provider operates, and average employment rate prior to study may help students judge 
the extent to which the course helped graduates find work. A limitation of this approach is that it 
puts the onus on students to synthesise the information and make judgments on how outcomes 
should be appropriately adjusted. While this may be a quick and easy option, it makes comparisons 
more difficult for students. 



Recommendations for developing the Student 
Outcomes Survey 

To adopt a ‘scoreboard’ framework for measuring quality, regardless of whether and how the 
measures are validated, we suggest a range of changes to the Student Outcomes Survey data 
collection. The recommendations below are ordered from highest to lowest priority, where 
recommendation 3 is likely to be the easiest to implement and 6 the most difficult. 

Recommendation 3: Include provider information in the collection 

Although information regarding the name and location of the institute is collected, as reflected in 
the Student Outcomes Survey cover letter sent to each person who completed recognised 
vocational training in the previous calendar year, this information is discarded and not made 
publicly available. With provider information included in Student Outcomes Survey dataset, a 
natural progression for the dataset would be to link it to administrative information collected by the 
institutes, such as information on the courses (such as their completion rates, attendance rates, 
contact hours, average class sizes) and information on the individual students (such as attendance 
records, whether study is full-time or part-time, expenditure per student, staff— student ratios, 
completion rates and academic history). This information would provide important information 
which could be used to put outcome information into context or used to validate the use of labour 
market outcomes. Linking the survey to provider administrative data may also provide a useful tool 
for analysing the factors that lead to favourable outcomes. 

Recommendation 4: Collect more information on students and their labour market outcomes 

While the Student Outcomes Survey currently asks individuals about earnings (Q29), it is 
categorical (eight possible categories ranging from ‘$1— $79 per week’ to ‘more than $1500 per 
week’) and not in its most useful form, especially if making comparisons across providers for a 
given course (where differences may be small). There are many examples of surveys that ask 
respondents to give their income in dollars, with high response rates, such as the Household 
Income and Labour Dynamics (HILDA) Survey and the Graduate Destination Survey (GDS). 

To be able to validate the use of outcome measures used as part of a ‘scoreboard’ approach, or at 
least to put the outcomes into perspective, more information on the nature of the students is 
required. For example, more information is needed on employment history, marital status, living 
and caring arrangements, place of residence (to control for differences in employment 
opportunities of graduates) and work experiences while studying. These factors may affect post- 
study employment outcomes, and to the extent that they differ across courses and providers, need 
to be accounted for when comparing post-study outcomes. 
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Recommendation 5: Increase the sample si%e and survey response rates 

For robust scoreboard information for specific courses across providers there needs to be a review 
of the current sampling method, in particular, a larger sample and a higher response rate. 

Recommendation 6: Expand the survey to include information on private fee-for-service courses 

Without information on private fee-for-service courses, it is difficult for prospective students to 
evaluate all the available options, especially students who may not be eligible for a publicly funded 
position, such as overseas students. Information on private fee-for-service is also required for 
public accountability, for example, reporting against Council of Australian Governments’ 
objectives, such as the proportion of graduates employed after completing training by previous 
employment status. 

Recommendation 7: Add a panel dimension to the Student Outcomes Survey 

At present, the Student Outcomes Survey attempts to measure the labour market benefits of 
training by using a single cross-sectional survey in the year following the completion of a module or 
qualification. While this is an economical way of evaluating the effects of training, it does not address 
an important related question: is employment obtained after training sustainable over time? 12 

One possibility would be to expand the one-off NCVER ‘Down the Track’ survey that followed up 
on the progress of respondents to the Student Outcomes Survey two years after the initial survey. 

In doing do, the Student Outcomes Survey would become a panel dataset conducted every one or 
two years, where the initial survey would elicit more contextual and historical information than in 
subsequent surveys. Subsequent surveys would focus primarily on employment and earnings, as 
well as job satisfaction. A large-scale panel dataset would capture the labour market dynamics of 
VET graduates more effectively. 

Although it might be considered that there would be some overlaps with HILDA and the 
Longitudinal Surveys of Australian Youth (LSAY), sample sizes of VET graduates in these two 
panel surveys are often small and not conducive to VET-specific analyses. There is a precedent of 
more focused panel data surveys being conducted. For example, the Department of Education, 
Employment and Workforce Relations (DEEWR) began a five-wave panel survey of income 
support recipients in 2006 (the Longitudinal Pathways Survey) because the sample size in HILDA 
is too small to allow detailed analysis of the labour market dynamics of income support recipients. 



12 This has also been identified as a weakness of data collection and research on VET in Hoeckel et al. (2008, p.28). 
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Conclusions 



Reforms in the VET sector are moving it towards a more demand-driven model of provision, one 
where funding is linked more closely to the choices of students. To ensure that such reforms 
deliver good outcomes for students and employers, it is important that students have adequate 
information on course quality across providers. Economic theory dictates that, in markets where 
there is poor information on quality of goods or services, there is a risk that agents will compete on 
price at the expense of quality (McMillan 2004; Kranton 2003), as evidenced by effects of market 
reforms in the health sector in the United Kingdom (Propper, Burgess & Green 2004). There is 
evidence that market reforms in the Australian VET sector during the 1990s have activated such a 
‘commoditisation processes’ (Anderson 2005). 

To avoid any possible deterioration in quality in the face of further reforms, namely, the 
introduction of the Australian Government’s VET FEE-HELP loan scheme, it is important that 
students are given adequate information on the quality of the courses provided by various 
providers. A recent OECD review on the Australian VET system emphasised this point and 
stressed the importance of making better use of the data from the Student Outcomes Survey 
(Hoeckel et al. 2008). According to Hoeckel et al. (2008), improving information on quality will 
enable prospective students to choose the course that best meet their needs. In this way, 
institutions are rewarded for the quality of their training and not just for reducing costs. 

In this study, we examined the possible use of the Student Outcomes Survey dataset to fulfil this 
role. Analysis of the data on student perceptions of quality in the survey found very little variation 
between courses and between providers. While there may be a number of reasons explaining this, 
including problems recalling aspects of the course six months after completion, we argue that post- 
study outcomes such as employment and wage rates are more meaningful for students when 
making choices between courses and providers and provide a better measure of course quality. 
Further, outcomes are also a measure of the value that employers place on the skills developed 
from the training. All else being equal, the more favourable the labour market outcomes from 
training, the more valued the skills are to employers. 

We recommend making information on student outcomes available as part of a ‘scoreboard’ of 
information on outcomes, courses and providers (recommendation 1), similar to the Good universities 
guide for prospective higher education students. Such a depository of information makes it easy for 
students to compare and contrast courses and providers. However, we acknowledge that such an 
approach does have its drawbacks. In particular, differences in the outcomes across providers may 
reflect differences in the regions and students serviced by providers, which may create perverse 
incentives for providers to bias their student intake, shift their location, or pressure poor students 
to exit prematurely. For this reason we suggest that raw outcome measures are validated against 
measures that control for differences in student characteristics and student opportunities across 
providers, such as output from regression models (recommendation 2). This would require 
additional information on student characteristics to be collected. 

Adopting a scoreboard approach with validated outcomes requires the alteration and expansion of 
the Student Outcomes Survey. From easiest to most difficult, these recommended changes are: 

•y" include individual provider information in the collection (recommendation 3) 

•y" collect more information on students and their labour market outcomes (recommendation 4) 
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■y" increase the sample size and survey response rates (recommendation 5) 

•y" expand the survey to include information on private fee-for-service courses 
(recommendation 6) 

add a panel dimension to the Student Outcomes Survey (recommendation 7). 
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Appendix A 



Table A1 Statements on the three aspects of course, Student Outcomes Survey 2005-08 

On a scale of 1 to 5, where 1 is strongly disagree and 5 is strongly agree, how, on average, would you rate the 
following aspects of the training? 

Teaching 

My instructors: 

1 . had thorough knowledge of the subject 

2. provided opportunities to ask questions 

3. treated me with respect 

4. understood my learning needs 

5. communicated the subject content effectively 

6. made the subject as interesting as possible 

Assessment 

7. I knew how I was going to be assessed 

8. The way I was assessed was a fair test of my skills 

9. I was assessed at appropriate intervals 

10. I received useful feedback on my assessment 

1 1 . The assessment was a good test of what I was taught 

Generic skills and learning experiences 

My training: 

12. developed my problem solving skills 

1 3. helped me develop my ability to work as a team member 

14. improved my skills in written communication 

15. helped me to develop the ability to plan my own work 

16. has made me feel more confident about tackling unfamiliar problems 

17. has made me more confident about my ability to learn 

1 8. has made me more positive about achieving my goals 

19. has helped me thing about new opportunities in life 
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Appendix B 

Figure B1 Scorecard for Justice Institute of British Columbia (from BCStats, 2008) 
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JIBC 


All BC Institutions 


Institution 


All BC 


Institution as % 






INDICATOR 












Provincial 


Standard 


as % of BC 


Institutions 5 


of BC Average 










Former Students 


Former Students 


Range 


Deviation 


Average 


Standardized 


(Standardized 










Value 1 


N 


Value 1 


N 


Min-Max 


Value 


(Index 3 ) 


Value 


Index) 






























In Applied Programs 


% 


94% 


276 


76% 


12,375 


32% - 100% 


19% 


124% 








c 

ns 


In Arts and Sciences Programs 


% 


6% 


17 


24% 


3,922 


0% - 68% 


19% 


24% 








a> 

0- £ 


In Applied Program, 3-6 Months 


% 


88% 


257 


17% 


2,799 


0% - 88% 


19% 


511% 








*— i 1 


In Applied Program, 7-12 Months 


% 


5% 


15 


31% 


5,075 


5% - 72% 


19% 


16% 








13 


In Applied Program, 13-36 Months 


% 


1% 


4 * 


27% 


4,337 


1 % - 47% 


11% 


5% 








O) 


In Arts and Sciences Program, Lower Division 


% 


6% 


17 


24% 


3,922 


0% - 68% 


19% 


24% 








CL 


In Applied Post-Baccalaureate Program 


% 


0% 


0 


1% 


164 


0% - 4% 


1% 


n/a 








Agriculture, Natural Resources and Sci.Tech. 


% 


0% 


0 


2% 


317 


0% - 1 5% 


4% 


n/a 










Arts and Sciences 


% 


6% 


17 


24% 


3,922 


0% - 68% 


19% 


24% 










Business Management 


% 


0% 


0 


14% 


2,273 


0% - 54% 


11% 


n/a 










Communications 


% 


0% 


0 


2% 


250 


0% - 4% 


1% 


n/a 








V) 

CO 


Computer and Information Services 


% 


0% 


0 


2% 


318 


0% - 6% 


2% 


n/a 








Construction and Precision Production 


% 


0% 


0 


10% 


1,567 


0% - 33% 


10% 


n/a 








< 


Education and Library Services 


% 


0% 


0 


5% 


736 


0% - 1 8% 


5% 


n/a 








E 


Engineering, Electrical and Electronics 


% 


0% 


0 


6% 


902 


0% - 16% 


4% 


n/a 








U) 


Health Related (see also Nursing) 


% 


76% 


222 


10% 


1,621 


0% - 76% 


17% 


762% 










Legal and Social 


% 


18% 


54 


5% 


881 


1 % - 53% 


11% 


341% 










Mechanical and Related 


% 


0% 


0 


6% 


1,035 


0% - 1 7% 


5% 


n/a 










Nursing 


% 


0% 


0 


4% 


700 


0% - 20% 


5% 


n/a 










Recreation, Tourism, Hospitality and Service 


% 


0% 


0 


6% 


1,027 


0% - 22% 


7% 


n/a 










Transportation 


% 


0% 


0 


0% 


68 


0% - 2% 


1% 


n/a 










Visual, Performing and Fine Arts 


% 


0% 


0 


4% 


680 


0% - 22% 


5% 


n/a 








V) 


Female 


% 


33% 


98 


54% 


8,788 


33% -81% 


11% 


62% 


71% 


47% 




E 1. 


Male 


% 


67% 


195 


46% 


7,502 


19% -67% 


11% 


145% 


29% 


229% 




S £ 


Age at Time of Survey (Years) 


Median 


31 


280 


24 


16,280 


22-43 


5.08 


129% 


32 


97% 




05 


Aboriginal % 




4% 


12 


5% 


775 


2% - 83% 


18% 


87% 


5% 


76% 




C 


Job Skills 


% 


55% 


161 


43% 


6,939 


23% - 56% 


10% 


128% 


53% 


105% 




c o~ 


Credential % 




18% 


52 


32% 


5,124 


18% -57% 


11% 


56% 


23% 


79% 




™ co o 

2 d) i: 


Credential and Job Skills 


% 


24% 


71 


21% 


3,447 


17% -31% 


3% 


114% 


23% 


106% 




“ w 


Other Reason 


% 


2% 


7 * 


4% 


580 


0% - 7% 


2% 


67% 


2% 


126% 




> 6 


Had previous post-secondary education 


% 


78% 


229 


47% 


7,594 


26% - 79% 


15% 


167% 


62% 


127% 




Q- S 


























> ■£ 


Of employed: 
























a 


Had Current Job Before/During Studies 


% 


59% 


168 


32% 


4,173 


20% - 59% 


10% 


184% 


34% 


173% 






Completed Requirements for Program Credential 


% 


98% 


282 


78% 


12,515 


52% - 100% 


16% 


126% 


94% 


104% 






Main Reason for Enrolling Met(4=Completely) 


Scale 4-1 


3.44 


291 


3.35 


16,076 


3.17-3.62 


0.11 


103% 


3.53 


97% 






Overall Satisfaction with Studies 2 


Scale 4-' 


2.97 


286 


3.36 


16,188 


2.97 - 3.60 


0.13 


88% 


3.40 


87% 






Write Clearly and Concisely (5=Very Well) 


Scale 5-1 


3.34 


161 


3.99 


13,505 


3.34 - 4.50 


0.22 


84% 


4.03 


83% 






Speak Effectively 


Scale 5-1 


3.56 


200 


3.97 


13,259 


3.56 - 4.39 


0.17 


90% 


4.06 


88% 






Work Effectively with Others 


Scale 5-1 


3.95 


279 


4.24 


15,447 


3.87 - 4.63 


0.17 


93% 


4.26 


93% 




05 *- 

a. co 


Analyze and Think Critically 


Scale 5-1 


3.93 


284 


4.22 


15,841 


3.93-4.39 


0.11 


93% 


4.23 


93% 




05 = 
I ■* 


Resolve Issues or Problems 


Scale 5-1 


3.74 


281 


4.05 


15,398 


3.74 - 4.38 


0.14 


92% 


4.09 


91% 




= « 


Use Mathematics 


Scale 5-1 


3.25 


209 


3.97 


11,746 


3.25 - 4.28 


0.24 


82% 


3.73 


87% 




g o 


Use Computers 


Scale 5-1 


2.97 


113 


3.90 


11,586 


2.97-4.14 


0.24 


76% 


3.71 


80% 




o > 


Use Other Tools & Equipment 


Scale 5-1 


3.92 


263 


4.12 


13,774 


3.81 -4.38 


0.19 


95% 


4.21 


93% 




Q 


Learn on Your Own 


Scale 5-1 


3.76 


264 


4.16 


15,668 


3.76 - 4.58 


0.17 


90% 


4.17 


90% 




Read and Comprehend Materials 


Scale 5-1 


3.59 


253 


4.17 


15,456 


3.59 - 4.45 


0.18 


86% 


4.16 


86% 






Quality of Instruction (5=Very Good) 


Scale 5-1 


3.97 


292 


4.23 


16,227 


3.97 - 4.53 


0.13 


94% 


4.28 


93% 




E 


Organization of Program 


Scale 5-1 


2.98 


292 


3.99 


16,184 


2.98-4.17 


0.25 


75% 


3.93 


76% 






Fair Assessments (Tests, Papers) 


Scale 5-1 


3.27 


289 


4.14 


16,207 


3.27 - 4.43 


0.23 


79% 


4.04 


81% 




'I £ 


Amount of Practical Experience 


Scale 5-1 


3.36 


292 


4.00 


15,373 


3.36 - 4.30 


0.23 


84% 


4.01 


84% 




o ® 


Textbooks and Learning Materials 


Scale 5-1 


3.40 


293 


3.86 


16,153 


3.40-4.17 


0.19 


88% 


3.92 


87% 




o ~ 


Library Materials 


Scale 5-1 


3.19 


178 


3.72 


12,151 


3.19-4.02 


0.19 


86% 


3.74 


85% 




to o 


Availability of Instructors Outside Class 


Scale 5-1 


3.29 


270 


4.15 


15,215 


3.29 - 4.48 


0.25 


79% 


4.06 


81% 




'■5 u 


Helpfulness of Instructors Outside Class 


Scale 5-1 


3.63 


273 


4.26 


15,438 


3.63 - 4.57 


0.20 


85% 


4.19 


87% 




to ® 


Quality of Computers and Software 


Scale 5-1 


2.81 


162 


3.92 


1 1 ,845 


2.81 -4.12 


0.27 


72% 


3.59 


79% 




< 


Quality of Other Tools and Equipment 


Scale 5-1 


3.35 


267 


3.96 


13,597 


3.35-4.18 


0.18 


85% 


3.92 


85% 






Total Number of Respondents 






293 




16,297 
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Standardized by Program Mix' 











JIBC 


All BC Institutions 


Institution 


All BC 


Institution as % 






INDICATOR 












Provincial 


Standard 


as % of BC 


Institutions 


of BC Average 










Former Students 


Former Students 


Range 


Deviation 


Average 


Standardized 


(Standardized 










Value 1 


N 


Value 1 


N 


Min-Max 


Value 


(Index 3 ) 


Value 


Index) 


Respondents 1 


-- Arts and Sciences Programs 






















C 

o 


Q. 

(/> 

f£ 


Currently Continuing Education 


% 


24% 


4 * 


77% 


3,010 


0% - 90% 


23% 


31% 


71 % 


31% 


o 

3 


6 


Taken Further Studies Since Leaving 


% 


29% 


5 * 


82% 


3,202 


0% - 90% 


22% 


36% 


82% 


36% 


LU 




Satisfaction with Transfer Experience (5=Very) 


Scale 5-1 


0.00 


0 


4.11 


2250 


3.33 - 5.00 


0.35 


n/a 


0.00 


n/a 


<5 


1 ?! 1 


Taken Further Studies at a BC Public Post-Sec. Inst. 


% 


60% 


3 * 


91% 


2,925 


33% - 100% 


20% 


66% 


91% 


66% 


t : 


H « 3 = 


Extent to Which Prepared for Further Study (4=Very) Scale 4-1 


3.20 


5 


3.38 


3,089 


3.20 - 4.00 


0.18 


95% 


3.38 


95% 


3 

U- 


5 i- u. m 


























_ Q. 


In the Labour Force 


% 


94% 


16 


72% 


2,793 


54% - 100% 


12% 


130% 


72% 


130% 




DC 


Employed 


% 


94% 


16 


66% 


2,570 


50% - 100% 


13% 


143% 


66% 


143% 






Employed in a Training-Related Job 


% 


88% 


14 


26% 


665 


0%- 100% 


27% 


338% 


26% 


338% 


CD 


a> 

>* 


Employed Full-Time, Training-Related 


% 


75% 


12 


14% 


354 


0%- 100% 


26% 


544% 


15% 


515% 


E 

>. 

o 


o 

a. 


Employed Full-Time, Non Training-Related 


% 


13% 


2 * 


31% 


808 


0%- 100% 


22% 


40% 


33% 


38% 


Q. 

E 


LU 


Employed Full-Time (30 hrs or more a week) 


% 


88% 


14 


45% 


1,163 


37% - 100% 


19% 


193% 


48% 


183% 


LU 




Employed Part-Time 


% 


6% 


1 * 


49% 


1,269 


0% - 60% 


18% 


13% 


52% 


12% 






Unemployed (of those in labour force) 


% 


0% 


0 


8% 


223 


0% - 25% 


7% 


n/a 


0% 


n/a 






Gross Hourly Wage (main job) 


Median 


$39 


13 13 


$ 


2,123 


$11 - $39 


$8 


304% 


$13 


304% 






Weekly Hours Worked (main job) 


Median 


38 


15 


25 


2,416 


20-40 


7 


152% 


25 


152% 


« » 
8 .22 

C "O 

3 3 
c n 


If 

511 


Usefulness of Training in Getting New Job (4=Very) 
Usefulness of Skills/Knowledge in Performing Job 


Scale 4-1 
Scale 4-1 


3.00 

3.86 


4 * 
14 


2.32 

2.40 


636 

1,155 


1.00-4.00 
2.07 - 3.86 


0.66 

0.56 


129% 

161% 


2.32 

2.40 


129% 

161% 


z> o 


£ ll 

























Total Number of Arts & Science Respondents 



17 



3,922 



All BC Institutions 



Respondents 1 -- Applied Programs 




X V) 
O Q 


Currently Continuing Education 


% 


20 % 


54 


28% 


3,488 


17% -43% 


8% 


69% 


17% 


117% 


DC 


Taken Further Studies Since Leaving 


% 


30% 


82 


34% 


4,185 


22% - 49% 


9% 


88% 


22% 


136% 




Satisfaction with Transfer Experience (5=Very) 


Scale 5-1 


3.44 


9 * 


3.91 


1188 


94%.25 - 4.16 


0.23 


88% 


3.66 




Jill 


Taken Further Studies at a BC Public Post-Sec. Inst. 


% 


56% 


46 


81% 


3,404 


50% - 92% 


10% 


69% 


67% 


84% 


o H “■ “ 


Extent to Which Prepared for Further Study (4=Very) Scale 4-1 


2.96 


70 


3.46 


3,875 


2.96 - 3.67 


0.16 


86% 


3.28 


90% 


Q. 

o S 


In the Labour Force 


% 


99% 


273 


90% 


11,112 


85% - 99% 


4% 


109% 


95% 


104% 


DC 


Employed 


% 


97% 


269 


85% 


10,439 


71% -97% 


7% 


115% 


93% 


105% 




Employed in a Training-Related Job 


% 


80% 


214 


81% 


8,404 


68% - 86% 


6% 


99% 


86 % 


93% 


"S 


Employed Full-Time, Training-Related 


% 


63% 


169 


68% 


7,080 


53% - 78% 


7% 


93% 


73% 


87% 


O 

Q. 

£ 


Employed Full-Time, Non Training-Related 


% 


19% 


50 


14% 


1,478 


7% - 22% 


4% 


131% 


12% 


151% 


LU 

6 


Employed Full-Time (30 hrs or more a week) 


% 


81% 


219 


82% 


8,563 


72% - 93% 


5% 


99% 


85% 


96% 




Employed Part-Time 


% 


7% 


18 


14% 


1,427 


4% - 24% 


5% 


49% 


15% 


44% 




Unemployed (of those in labour force) 


% 


1% 


4 * 


6% 


673 


0%- 19% 


5% 


24% 


2% 


66% 




Gross Hourly Wage (main job) 


Median 


$22 


204 19 $ 




8,838 


$17- $27 


$2 


119% 


$20 


108% 




Weekly Hours Worked (main job) 


Median 


40 


233 


40 


9,937 


35-40 


2 


100% 


36 


111% 


5* E 


Usefulness of Training in Getting New Job (4=Very) 


Scale 4-1 


3.38 


89 


3.47 


6,322 


3.14-3.65 


0.14 


98% 


3.67 


92% 


° Q.= 
C 3 


Usefulness of Skills/Knowledge in Performing Job 


Scale 4-1 


3.21 


219 


3.39 


8,536 


3.15-3.58 


0.13 


95% 


3.49 


92% 


£ u- 


























Total Number of Applied Respondents 






276 




12,375 














All BC Institutions 



* Relatively small sample size, interpret with caution. 

1 Respondents are former students who have completed all or a significant portion of their program. TRU-Open Learning submits only those students who have fully completed their program. 

2 This is a different response scale from previous KOI reports and results cannot be compared (see Section A for more details). 

3 The "Index" is the institution value divided by the provincial value and multiplied by 100. Index values greater than "100%" indicate a higher than average institution value. Index values are "n/a" if there are no respondents 
in the category. All values are rounded, although calculations are done using several decimal places. 

4 The data in these two columns are calculated the same way as the data in the un-highlighted columns (All BC Institutions Value and Index), except that the program mix of JIBC is used to weight the All BC Institutions 
Standardized Values. The Standardized Index therefore allows an analysis of how JIBC is performing compared to the rest of the institutions in the system , if they were to offer the same program-mix as JIBC. The 
methodology used to calculate these program-mix standardized variables can be found in Appendix 2. 





















